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ABSTRACT 



What needs to be presented to practitioners, and particularly, school leaders in 
training, so that they can become “educated consumers” of education research? 
This paper attempts to sort through the issues and evaluate some of the 
instructional alternatives available to education administration faculty and 
curriculum designers. Taking into account a political and intellectual context that 
features tenacious and persistent debates about identifying quality research, this 
paper employs a three-pronged approach to analyzing the problem of introducing 
research literacy to educational leaders. First, placed in the context of course 
construction, a consideration of recent commentary on how best to introduce 
practitioners to education research provides an overview of debates among faculty 
in Schools of Education. Second, a content analysis of three popular textbooks on 
educational research methods allows for observations about the quality and 
applicability of instructional resources available to methods teachers. The paper 
also incorporates data collected from surveys of teachers and administrators in 
two graduate educational administration programs for future educational leaders 
located at a prominent research-oriented School of Education. 
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“/ guess my biggest problem with the class is that I don ’t see how this will help me 
become a better administrator. Knowing what I know from my 20+ years in independent 
schools - 1 know it would be impossible for me to conduct research as an administrator. 
It’s not that I don’t see value in research: I do. It’s just that it is not applicable to us 
because there is simply not enough time in the day... ” Graduate student reflecting on the 
utility of an introductory course on research methods in education, October 2002 



What kind of curriculum in research methods needs to be presented to 
practitioners, and particularly, school leaders in training, so that they can become 
educated consumers of education research? This paper attempts to sort through the issues 
and evaluate some of the instructional alternatives available to education administration 
faculty and curriculum designers. Preparing tomorrow’s leaders of schools for their roles 
as managers of complex organizations can take on many forms, but a very common 
training forum is the use of graduate level courses in education administration in Schools 
of Education. Programs for practitioners usually include at least one class that focuses on 
the methodologies aligned with education research. It is a concern with the instructional 
content of these courses that this paper focuses upon. 

As graduate students, most teachers and administrators in programs dealing with 
educational administration and leadership will be asked to produce academic style 
research for their courses and major degree requirements. But beyond the academic 
necessity, with the passage in 2001 of the Elementary and Secondary Education Act by 
Congress, commonly known as the “No Child Left Behind” Act, school leaders have had 
to come face to face with federal legislation that requires them to marshal “scientific 
evidence” to back-up their curricular and programmatic choices. It seems that it is more 
important than ever to make sure that district and building- level leaders in schools have 
the skills to be “educated consumers” of education research. Ideally, this means that 
school leadership needs to be able to identify what is really research rather than rhetoric 
and have the capacity to look at the evidence presented - the narratives, statistics and 
pictorials — with caution, and even skepticism. To be able to see through eloquent words 
and politics, recognize the claims being made, and be in a position to judge the strength 
of the argument and the pertinence of the evidence being marshaled, and to do this by 
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sifting through the almost overwhelming flow of education research produced in the 
United States, is not a task for the unskilled. 

Taking into account a political and intellectual context that features tenacious and 
persistent debates about what is good research, this paper employs a three-pronged 
approach to analyzing the problem of introducing research literacy to educational leaders. 
First, a consideration of recent commentary on how best to introduce practitioners to 
education research provides an overview of debates among faculty in Schools of 
Education. Second, a simple content analysis of three popular textbooks on educational 
research methods allows for some observations about the quality and applicability of 
instructional resources available to methods teachers, as well as for the laying out of some 
alternative organizational strategies for that material. The paper also incorporates data 
collected from surveys of teachers and administrators in two graduate educational 
administration programs for future educational leaders located at a prominent research- 
oriented School of Education (n=53). The feedback provided by these students on the 
spectrum of research methods presented to them during an introductory course to research 
methods and their assessment of the relative usefulness of this information provides 
insight about what future educational leaders themselves think they should know; another 
key set of considerations for those who wish to engage and teach them. 

What Do Education Leaders Need to Know? 



As an instructor in a course that introduces research methodologies to 
practitioners who have been singled out as potential school leaders, I have found it a 
challenge to determine what kind of curriculum best serves the needs of my students. 
Methods courses have an understandable reputation as being tremendously boring; that is 
understandable since teaching methods means teaching about abstract notions about 
knowledge production, claims and evidence and introducing some very complex 
techniques in an understandable manner. But beyond that evident instructional challenge 
lies another more insidious issue about curriculum design. Making the case for the 
relevance of understanding research methods is, to my mind, an integral part of effective 
instruction. Should an introduction to methodologies be a scoop up and focus exercise on 
a few, well-chosen and practitioner-oriented methods? Or should it skim the surface of 
the wide variety of methodologies in practice in education research? My stance has been 
that providing a broad overview of possible research methods, along with the basic 
vocabulary of social science research, a review of research resources and an emphasis on 
research ethics is what future education leaders need to know. However through my 
classroom experiences, in discussion with colleagues, and with quite a bit of personal 
reflection, I have begun to question whether or not a broader curricular approach serves 
the purpose of making teachers and administrators into educated consumers of research, 
or if it instead does the opposite: simply convinces future leaders that research is 
irrelevant to their experience and careers as educators. 

The overview approach to teaching research methods is not without its own 
problems. Providing skills and thereby producing discriminating consumers of research 
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through courses in Schools of Education is complicated by the fact that methods are tools 
for understanding that are built on rather abstract notions about the way we know things 
and the way we organize knowledge. Moreover, the field of education research itself 
encompasses a spectrum of research traditions and approaches that span the social 
sciences, dip into the humanities and more than brush against several important 
professional fields like social work and law. My assumption has been that to become a 
discerning consumer of research one needs to know the norms of research: the vocabulary 
of a field, the techniques available to build inquiry; and the resources that enable us to 
assimilate the work of predecessors and colleagues. But what if those simple building 
blocks are either hotly contested or non-existent as is the case of education research? 

The field of education is a big tent that includes anyone that conducts any kind of 
research that deals with instruction and learning, making it an inherently interdisciplinary 
and even eclectic pursuit. The field’s inclusiveness is a strength - interdisciplinarity 
means the sharing of ideas, findings and methods between disciplinary traditions - and 
sharing produces new and innovative ways of knowing things that strict adherence to a 
particular disciplinary tradition might not yield. But being a collection of disciplines 
rather than a single, coherent, field of study has major drawbacks. Constituencies 
involved and trained in very different research traditions - among them, philosophy, 
psychology, anthropology, economics, sociology and history, not to mention applied 
fields such as social work, clinical psychology, and policy studies - do not share common 
understandings, vocabulary and methods. They do not ask questions or measure results in 
the same way. 

Education thus has not developed a “core” of knowledge - “what every good 
scholar should know” (Schoenfield, 1999). Nor does it have a selection of theory or 
research literature that serves as the basis for universal understanding of how one 
conducts research - in other words, a “canon.” This makes it very difficult if not 
impossible for knowledge to accumulate and build on itself Rather than constructing new 
ideas based on the foundations of preceding work, educational researchers are stuck in 
what David Labaree evocatively calls “quonset huts of knowledge” (Labaree, 1998) - 
short term or ad hoc groupings of like-minded researchers pursuing understanding but not 
connected to previous research or others doing similar work. Fragmentation of research 
results from the often self-imposed isolation and specialization of educational 
researchers. 

The consequences are that training educational researchers utilizing a 
comprehensive overview perspective is very difficult: any attempt to introduce core 
knowledge and key literature requires covering a wide, maybe too wide, spectrum of 
styles of inquiry. As discussed in depth by Pallas (2001) the selection of what 
epistemologies and methods are to be covered in the graduate classroom is a process that 
inherently privileges certain kinds of epistemological traditions over others. An 
introduction to research methods means acknowledging that the course is as much a 
presentation of a selected number of social science methodologies as it is about research 
on education. Graduate students in education are attending courses in research methods to 
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be literate in their field. But skimming the surface of a series of prominent social science 
research methods may be simply confusing as well as make students feel daunted in terms 
of their ability to internalize, retain and utilize these methods themselves. This is 
particularly true for single semester methods courses. Successful curriculum in methods 
should be able to manage the line between necessary content and potential information 
overload. 

Concern with curricular content has elicited some recent debates among scholars 
in Schools of Education. I have framed the issues surrounding content and responsiveness 
to student needs into two: methodological hierarchies and academic versus practitioner 
orientations. After a brief discussion of these themes, the paper presents some preliminary 
empirical analyses, namely a content analysis of three popular graduate-level research 
methods textbooks and an examination of graduate student responses to an introductory 
research methods course. 

What Should be of Concern to Methods Instructors? 



Methodological hierarchies 

The “No Child Left Behind” Act of 2001 did not arise from an intellectual 
vacuum: within the academically-oriented education research community and funders of 
that research, strong, publicly-voiced ideas about what constitutes good research abound 
and are germane to considering what is worthy of being taught to future leaders of 
education. Clearly reflected in that legislation is a view shared by many distinguished 
researchers in the field that education research should be more “scientific” in its conduct 
and analysis (National Research Council 2002; Mosteller and Boruch 2002). Moreover, 
the Department of Education under the Bush administration has made strong 
pronouncements that there is a gold standard in research design, namely, that the highest 
quality research is composed of experimental designs that use randomized controlled 
trials. This stance has been backed up by congressional appropriations and organizational 
reform of the Department of Education in the summer of 2002 that mandates that federal 
dollars must support this “highest level” of rigorous research. Ignoring generations of 
epistemological debates within the academic community, the U.S. government has 
sanctioned a de facto hierarchy of methods regarding scholarship in education; a fact that 
has not been disregarded by the educational research community (see Educational 
Researcher, Special Issues, Vol. -,Winter 2003). 

The most rigorous science falls imder a category variously labeled “formalist”, 
“quantitative” or more properly called, “positivist.” Approaches to research design and 
methods of this sort encompass experimental, quasi-experimental and rigorous survey 
research methods, among others. Positivism rests largely in a legacy of inquiry about 
education that derives from the fields of psychology and sociology and the sub-field of 
educational measurement and statistics. The scientific approach to understanding learning 
was influential in the development of Lf.S. teacher education throughout the first part of 
the 20*'^ century (Schneider, — ). The impact remains today: a quick review of a 
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selection of education research methodology textbooks that are utilized in introductory or 
survey courses on education indicates that all were written by authors with positivist 
orientations (Ary, Jacobs and Razavieh; Fraenkel and Wallen; Wiersma, Best and Kahn; 
Langenbach, Baughn and Asgaard). 

However, positivist approaches to education research have never gone 
unchallenged. (See Greene, 1997 for a particularly passionate and well reasoned argument 
against the structure of this kind of methodological hierarchy.) Ongoing discussions about 
what constitutes good science beyond positivist approaches and methods are also being 
pursued in a variety of public and academic fora, such as the continuing efforts of the 
National Research Council to investigate the incentives that improve the structure of 
education research, joint work by the Social Science Research Council and the National 
Academy of Education focused on mapping what constitutes quality in education research 
within and outside of the academy, and efforts by the Campbell Collaboration to begin to 
collate meta-analyses on key education questions and issue areas. 

The implications of the current politics of methods for courses designed to engage 
educational leaders are several. First, the fact that there is a politics of methods and that 
there are large resource allocation issues attached to it is important for students to 
understand. Methods might be built on abstract notions of what knowledge is, but the 
ongoing struggle about what constitutes quality research is contemporary, fast-breaking 
news and interesting to discuss. As I indicate in my lectures, choosing methods is often 
seen as choosing sides in a battle not just for academic bragging rights, but for the 
creation of particular kinds of research agendas with real dollars at stake. 

Second, the epistemological divides that set up methodological rivalries must be 
made to clear in order that students better understand the context of what is happening to 
education research today. A basic description of the spectrum of epistemological 
possibilities and the research methods derived from them is vital. Students need to realize 
that accepting a methodological hierarchy has consequences; namely, the potential 
exclusion of certain kinds of knowledge and information about learning and instruction 
(Weiss, 2002). They can only do that if they realize what is being suggested by exclusion 
by whom. I ask them to consider if the loss of education’s “big tent” might prove to be a 
larger loss than we can bear since it not only suggests that education research may only be 
for a particular kind of researcher, but that there will be whole categories of inquiry that 
could provide solutions or prove useful in our analysis of problems in the classroom or, 
more broadly, in the politics of schooling that will no longer find funding or favor among 
scholarly societies and publications. 

Third, and hopefully quite obviously, providing an evenhanded approach to the 
description of the range of methods available to researchers in education is key. Both the 
benefits and the drawbacks associated with these tools of inquiry, should be made as plain 
as possible. To be fair, some personal reflection about stances should be thrown in. As 
no faculty member of any School of Education can or should claim to be an objective or 
impartial about such matters, it is necessary to be up front about one’s own personal 
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methodological training and predilections. This can elicit some interesting classroom 
dialogue. 

Finally, emphasizing the evolving and additive nature of methods is another 
important and potentially engaging theme. Pallas (2001) has noted that teaching 
epistemological diversity is a problem that is not going away: rather, with the rise of new 
approaches to knowledge production and research design, understanding that diversity is 
more important than ever. Chronicling the development of more contemporary methods 
in the classroom, for example the rise of postmodernist approaches to research and the 
reactions and counter-reactions to it, I have found, provokes students to think about the 
flow of innovation and improvement in methods, as well as reinforces their understanding 
of the deeply contested nature of what marks good research. 

Academic v. practitioner-focused research 

Beyond the debates about the relative worthiness of various research methods 
must be overlaid another issue of concern: how does practitioner-oriented research of the 
kind that future education leaders will most often run into and practice in graduate 
classrooms fit into the spectrum of methods that education leaders must learn to 
understand? To some extent all graduate students in education administration degree 
programs at Schools of Education share this common dilemma: they are required to 
become producers of education research during their tenure as students that is judged by 
academic standards and precepts, but in the real world, as teachers and administrators, 
their ability as critical readers of research is much more important. Graduate students, 
particularly those who are being prepared to serve as educational leaders, are strung up 
between the poles of academic and practitioner-centered research. As Anderson (2002) 
has noted, this has led to a kind of “schizophrenia” of outcomes: research products such 
as dissertations that attempt to meet academic research standards but with content that 
continues to reflect insider knowledge and beliefs and the shifting, complex reality of 
particular school and educational settings. 

Suggesting that academic research methods are relevant to the real world means 
confronting those strongly held beliefs of teachers and administrators turned graduate 
students. Their “personal epistemologies” are based on what they have experienced in the 
real world of the classroom, faculty meetings and parent-teacher conferences (Neumann, 
Pallas and Peterson, 1999) and can engender resistance to discussions of more abstract 
and theoretical understandings of the various social, political and economic phenomena 
involved in understanding schools and schooling (Ibid., Labaree, 1998). Moreover, for 
these students research relevance is often signified by findings rather than the process 
through which those findings were made. My students want to know which study 
provides definitive evidence and answers for problems they find in the classroom or 
among their administrative colleagues. The fact that the multiple iterations and variation 
in conditions of research suggests the right answer is “it depends” is very frustrating to 
them. Understanding the nuances of what “it” means does not seem particularly 
empowering. 
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Adding practitioner-oriented methods such as Action Research to course coverage 
is an obvious response to this concern. But again, it is important to practice transparency 
in presentation. The fact of the matter is that practitioner-oriented methods are not 
considered to be “real” or “legitimate” research, despite some very hefty academic 
legacies, such as Kurt Lewin’s emphasis on problem solving and postmodernism’s 
reification of the personal narrative (Metz and Page, 2002). The low status of research 
pursuits by education professionals seems in keeping with the low status generally 
accorded to teachers and administrators in the United States, and is an excellent subject 
for classroom discussion and debate. 

A similar topic for discussion concerns the distinctions to be made between 
academic, peer-reviewed journals and online resources and the publications produced by 
teacher associations and other education oriented organizations that are much more 
familiar to education leaders turned graduate students of education. Being able to 
recognize when an article is actually research rather than journalism or even advocacy is a 
key exercise. I have assigned students the task of finding a research article to critique with 
a prerequisite that they bring it to me for a check on whether or not it is actually research 
before they begin their analysis. Following that exercise is a class discussion of the 
“markers” of academic publications. They also are asked to identify at least two academic 
research journals that deal with their area of specialty as educators by going to the 
university library and interviewing faculty. These may seem to be rudimentary pursuits 
and perhaps unworthy of graduate level education, but I have found them to be a means of 
re-orienting practitioners and giving them some baseline guidance on how to sort through 
the plethora of resources available to them. 

The tension between academic and practitioner-oriented approaches generates 
another, perhaps obvious, implication for methods courses. The fullest possible 
integration of real world education problems and case studies is an important means to 
illustrate the usefulness and relevance of more traditional tools of research. This can be 
delivered via short simulations or problem-solving modules centered on using particular 
methods and by assigning the execution (or partial execution if time disallows it) of a 
small-scale research project. The point is that when conducting real research in the real 
world things get very messy. An introduction to methods presenting the paradigms of 
quality and the ideals of validity and reliability presents what might seem to be 
impossibly high standards for graduate students to meet. Students’ notions of the 
applicability and utility of research methods can be better shaped if they are asked to 
plunge in and experience for themselves how tools of research can be applied and what 
kinds of insights may or may not be gained from them. 

Finally, and quite related to the notion of distinctions between academic and 
professionally produced research, is the issue of audience. Who are researchers writing 
for and how do they communicate what they know? Since the goal of most of education 
research and all of practitioner oriented work is problem solving, it makes sense that 
being prepared to represent research findings to multiple audiences - scholarly. 




9 



8 Rough draft - Not for citation without permission of the author 



practitioner, policymakers, the lay public - and being able to take into account how 
writing for those audiences can shift the tone and content of research findings, is a very 
important for graduate student to understand in both their roles as producers and 
consumers of research (Neumann, Pallas and Peterson, 1999). We are hopefully not being 
put in the position of instructing future education leaders to restrict their writing style and 
presentation to the use of often arcane language and rhetoric. There is a balance to be 
struck in dissemination as there is in execution of research. In exercises requiring 
students to critique research articles, I enclose a rubric that prominently features these 
kinds of questions. Who is the author writing for? How does the nature of the audience 
shape the piece? Is there evidence of a political or social agenda being pushed? What 
might be some of the underlying and unarticulated questions that are being responded to? 
Are there contextual factors such as the journal selected, timing of publication that are 
part of the explanation of why the article asks certain research questions, uses certain 
kinds of data and does or does not contest previous research? Similar questions are posed 
about their own work. 

Responsiveness to the issues presented by methodological hierarchies and 
academic/practitioner approaches can help shape a methods course, but how usefiil are 
current textbooks in support of an overview style curriculum? And what about student 
reaction to this set of curricular choices by the instructor? It is to these questions that we 
now turn. 

Methods and Data 



Methods Textbooks 

A purposive sample of textbooks available for introductory courses in education 
research methods was drawn using the following selection criteria: attention to reasonably 
recently produced or revised volumes (within five years); texts with long-standing 
popularity as evidenced by there being four or more editions produced; and texts that are 
from different publication houses. Two of the three textbooks examined (Fraenkel and 
Wallen; Wiersma) were utilized by the researcher in course instruction. The three 
textbooks selected for analysis are listed in Table 1. 



Table 1 : Introductory Level Textbooks on Education Research Methods 


Author(s) 
Editions Rev. 


Title Publisher No. of 


Year Last 


Ary, Jacobs, Razavieh 


Introduction to Research in Ed. Wadsworth 6 


2002 


Fraenkel and Wallen 


How to Design & Evaluate Research in Education 

McGraw-Hill 4 


2000 


Wiersma 


Research Methods in Ed. Allyn and Bacon 7 


2000 



The analysis entailed the examination and comparison of chapter organization and 
contents and page allocation for each book along the following dimensions: the inclusion 
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of description of the field of education research; the presence or absence of an 
introduction of theoretical (and particularly epistemological) concerns in research; the 
presence or absence of a description of steps in research design; a count of the number 
and variety of methods included and excluded and the balance struck between 
quantitative and qualitative methods; and the inclusion of case studies and examples of 
those methods in use. 

Student Surveys 

Short surveys were distributed to two cohorts of graduate students in education at 
a Research I University in the northeast United States in the summer and winter of 2002. 
Copies of the survey are in Appendix 1 . The cohorts were affiliated with two different 
competitive postgraduate degree programs in educational administration designed for 
teachers and administrators who are promising future education leaders. The first cohort 
(Cohort A) was selected from teachers and administrators in public schools in a suburb of 
New York City; the second (Cohort B) was selected from an international applicant pool 
of independent school teachers and administrators. Both cohorts were required to take a 
course in introductory research methods as part of their degree requirements. Each cohort 
received the same curriculum from the same instructor. Potentially significant differences 
in course content and conditions beyond the make up of the cohorts themselves included 
the fact that Cohort A was instructed in a compressed three week period during the 
summer while Cohort B had a full semester length period of instruction. Cohort A had 
already cycled through several compressed semesters of courses, while Cohort B was 
working through its first semester of instruction. Textbooks assigned for the two cohorts 
were different, although additional assigned reading material was the same. Class 
assignments were also different in light of the very different amount of time available for 
instruction.' 

Findings 



Methods Textbooks 

The results of the rudimentary analysis along the dimensions of attention to 
relevance, balance of coverage and attention to key topics can be seen in Table 2. Of 
particular interest are the two dimensions that attempt to get an indicator of the 
integration of methods instruction with relevance to the field of education, namely, 
whether or not there is a discussion or description of education research as an undertaking 
or as a field of study included in the textbook and whether or not there are case studies 
and concrete examples of methods use integrated within the text. All three volumes pay 
scant attention to the job of describing the idiosyncrasies and challenges associated with 
education research, although Fraenkel and Wallen do finish their volume with a special 



' Cohort A used Fraenkel, J. and Wallen N. “How to design and evaluate research in education” and was 
assigned two article critiques and a research project proposal. Cohort B used Wiersma, W. “Research 
methods in education” and was assigned two article critiques, a library exercise, a group research project 
proposal and the completion and presentation of a group research project. 
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section on doing research in school settings. Not surprisingly, all three textbooks pay 
attention to the integration of examples from education research, although the amount of 
space allotted to longer examples and case studies varies quite a bit. Fraenkel and Wallen 
provide the most consistent and comprehensive set of examples, particularly in the 
chapters covering specific methods. Ary, Jacobs and Razavich provide short references 
and examples, but with the exception of a full study exemplifying qualitative analysis, 
devote little attention to education examples when describing the various methods. 
Wiersma consistently uses short examples of education research throughout the text. 

Whether or not the texts examined privileged quantitative orientations is relatively 
difficult to surmise from this limited data. We can see that the amount of statistical 
instruction included in all three is about the same amount in terms of chapter counts and 
that there are always more chapters devoted to descriptions of quantitative methods than 
qualitative, although that ratio varies either by chapter count or page count. Once again, 
Fraenkel and Wallen stands out with their relative balance of coverage and their special 
attention to education practitioners and their research pursuits. 

In terms of two specific areas of focus, attention to epistemological and theoretical 
concerns, and coverage of research design steps, there is also divergence. Ary et. al. focus 
a large part of the first chapters of their text on describing the philosophy, constructs and 
theories around knowledge building. They also devote approximately 30% of their 
chapters to the various steps entailed in producing research, from development of the 
research question to dissemination and critical reading. Fraenkel and Wallen give very 
short shrift to abstraction and historical context, but provide the most extensive coverage 
of research design steps, utilizing 39% of their total chapters on the subject. Wiersma 
pays almost as much attention to theoretical and epistemological concerns as Ary et. al, 
but only devotes 25% of his chapters to the research design process. 

Rudimentary analysis can only suggest rudimentary findings. The lack of 
consistent attention to drawing in and engaging educators is striking. The nature of the 
enterprise itself, especially its politics, the relevance of methods to pressing educational 
questions and the larger epistemological and theoretical debates are not major emphases 
of these texts. On the other hand, the building blocks of research design are clearly and 
appropriately the major focus of the volumes. Finally, the amount of pages allocated to 
positivist approaches and statistical instruction is consistent with conventional wisdom, 
that is, that quantitative issues get consistently more coverage than qualitative matters. 
However it should be acknowledged that the complexity of positivist methods necessitate 
careful explanation of numerous tools of analysis as well as specific vocabulary and 
procedures, and thus perhaps the allocation of space is justifiable and should not be 
attributed to author or field bias. Tentative generalizations to be made from this data 
suggest that the goals of comprehensiveness coverage seem generally well served by 
these texts, although the mission of relevance seems to be more inconsistently handled. 
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Table 2: Textbook Analysis 






Dimension 










Ary, Jacobs and 
Razavieh 


Fraenkel and 
Wallen 


Wiersma 


Description of the field of 
education research? 


1 page at end of 

introductory 

chapter 


1 page, first page, 
but chapter on 
during research in 
schools at end (10 
pages) 


2.5 pages at end 
of introductory 
chapter 


Attention to theories and 
epistemologies 


Yes: 1.5 chapters 


Yes: brief 
consideration for 2- 
3 pages 


Yes: most of 1 
chapter 


Coverage of Research Design 
Steps 


2 sections (5 
chapters) + part of 
final chapter on 
reporting : 30% of 
chapters 


2 sections (9 
chapters total): 
39% of chapters 


4 chapters:25% 
of chapters 


Number of Methods Featured 
as Chapters 


4 quant 
2 qual 


5 quant 
4 qual 


3 quant 
2 qual 


Quantitative/Qualitative 
Balance (number of text 
pages) 


144 pages quant 
68 pages qual 
ratio = 1:2.1 


1 84 pages quant 
139 pages qual 
ratio = 1 : 1 .3 


168 quant 
69 qual 
ratio = 1:2.4 


Use of Case Studies and 
Examples from Schools and 
Educational Research 


Short examples 
from studies 
throughout text. 
Single study as 
example of 
Qualitative 
Research 


Lengthy sample 
studies throughout 
text. Full section 
for practitioners. 


Short examples 
from studies 
throughout text 


Amount of Statistical 
Instruction 


2 chapters (One 
section) : 12% of 
chapters 


3 chapters: 13% of 
chapters 


2 chapters: 
12.5% of 
chapters 


Total Chapters 


17 (Six sections) 


23 (Seven sections) 


16 


Total Pages of Text 


533 


642 


415 



Student Surveys 

In analyzing the results of the student surveys, the sample was stratified along two 
dimensions of interest: by the public or private nature of the school system the student 
came from, and by the professional category the student claimed: either as a teacher, 
administrator or combined teacher/administrator. The second order concern in both 
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circumstances was whether or not the student had taken previous courses in methods, thus 
belonging to either a novice or experienced category. 

Table 3 indicates the breakdowns for the public/private and professional category 
dimensions, as well as the novice/experienced category. 



Table 3: Student Survey Sample Sizes by public/private school distinction, 
professional category and novice/experienced status 




Number of 

Students 

(n=53) 


Novice in 

Methods 

Courses 


Experienced in 
Methods Courses 


Public School 
Background 


26 (49%) 


31% 


69% 


Private School 
Background 


27 (51%) 


67% 


33% 


Teachers 


25 (47%) 


44% 


56% 


Administrators 


10(19%) 


40% 


60% 


Teacher/Administrators 


12(23%) 


67% 


33% 


Other Professionals 


6(11%) 


50% 


50% 


Novice Students 


26 (49%) 






Experienced Students 


27 (51%) 







The sample broke down fairly evenly in terms of public/private school background and 
novice and experienced status with methods instruction. The highest percentage of 
students previously exposed to methods courses were groups with public school 
backgrounds (69%) or those who identified themselves as teachers or administrators 
(56% and 60% respectively.) The groups with the lowest exposure were 
teacher/administrators (33%) and those with private school backgrounds (33%). 

The survey itself had four parts: 1) a brief section of demographic descriptors; 2) a 
Likert scale series on the usefulness of the various elements of the methods course, 
including class lectures and activities, readings, and assignments; 2) a check off list of 
methods from which the students were asked to indicate those that they had developed a 
preference for using; 3) a check off list of methods from which they were asked to 
indicate which they would like to learn more about; and 4) a question about whether or 
not they would like to take further courses in those methods with an open ended response 
opportunity. 

The core questions to be answered from the surveys included the following: 

1 ) Which of the five groups identified was the most resistant to being instructed in 
methods? This issue area was broken down into several sub-sets; 

How did the groups break down in terms of developing preferences for 
methods? 

How did the groups break down in terms of wanting to acquire more 
knowledge about methods? 
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2) Did elements that the instructor thought important to include in the course, such as 
exposure to academic publications and creating the capacity among students to become 
critical readers of research, seem useful to the students? 

3) Did these future education leaders prefer the more practitioner-oriented methods 
presented? What were the most popular methods among those that had preferences? 

Resistance to instruction, derived from questions dealing with methods preference 
and intentions to learn more, resulted in the following frequencies: 



Table 4: Resistance to methods instruction by public/private school distinction, 
professional category and novice/experienced status 




Developed 
methodological 
preferences? 
Answered No 


Want to learn 
more about 
methods? 
Answered No 


Interested in 
taking more 
methods 
courses? 
Answered No 


Public School 
Background 


58% 


15% 


46% 


Private School 
Background 


33% 


33% 


44% 


Teachers 


32% 


32% 


52% 


Administrators 


60% 


20% 


50% 


Teacher/Administrators 


50% 


17% 


58% 


Other Professionals 


66% 


0 


33% 


Novice Students 


54% 


23% 


54% 


Experienced Students 


37% 


22% 


48% 



The majority of students in all categories want to learn more about methods, indicating 
general opermess to learning, but the students’ identification of a method of preference, 
which is a crude indicator of their degree of engagement with or focus on the domain of 
methodology, is quite a bit more varied. Teachers and private school background 
students in the sample are more strongly inclined to state their methodological 
preferences, as are experienced students. Graduate students from public schools, 
administrators, other professionals and novice students are more reluctant to indicate their 
preferences. There is also some uniformity concerning interest in taking additional 
courses. Except in the case of other professionals, about half of all respondents said that 
would not be interested in doing so. It is important to note about that result, however, that 
interest and opportunity to select another methods course might have been confounded in 
the survey. Written comments indicate that for many students there is a feeling that there 
is inadequate time or credit allowance for an additional methods course in their degree 
program structure. “I would if I could” seemed to be the message for a number of 
respondents. Given these results, in terms of public versus private, overall, private school 
students were more resistant to method instruction. Administrators, 
teacher/administrators and other professionals seemed to be more aware of their 
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methodological preferences, and more willing to take on additional instruction than 
teachers. Similarly, novice students were less engaged than experienced students, but 
both groups were similar in their interest and intent to learn more. 



As mentioned, several assignments given to these cohorts were seen by the 
instructor as being crucial in terms of promoting knowledge about the sources of research 
and gaining capacity in critical reading. Did the students find those course elements to be 
valuable? Students were asked to use a 1 -4 Likert scale to indicate whether or not 
particular course elements were of use to them. Table 5 indicates reactions to the 
publications exercise in which students were asked to inquire and select academic 
publications, and a journal critique exercise, which required students to produce an essay 
that critiqued the methodological choices and presentation in particular academic journal 
articles. 



Table 5: Ratings of instructor emphases as useful by students by public/private 
school distinction, professional category and novice/experienced status (Scale 
1 -4; 4 highest; simple averages) 




Rating of 

Publications 

Exercise 


Rating of 
Critical Reading 
Capacity 
Exercise 


Average Score 
of Two 
Exercises 


Public School 
Background 


3.15 


3.08 


3.12 


Private School 
Background 


1.77 


2.93 


2.35 


Teachers 


2.4 


3.25 


2.83 


Administrators 


1.45 


3.15 


2.30 


T eacher/Administrators 


2.83 


2.84 


2.84 


Other Professionals 


3.0 


2.58 


2.79 


Novice Students 


2.35 


3.14 


2.75 


Experienced Students 


2.58 


2.95 


2.77 



The results indicate that students were generally much less enthusiastic than the 
instructor about the need to take on these assignments. The critical reading exercise was 
(with the exception of the miscellaneous professionals) considered to be more useful than 
the publications exercise by the students. Public school background students, teachers and 
administrators thought fairly highly of the critical reading assignment. Public school 
background students in general liked both assignments. Private school background 
students and administrators found the publications exercise to be less than useful. Both 
of these groups found the two exercises to be less helpful than their counterparts. 
Novice/experienced status did not seem to cause much variation in this set of 
comparisons. 
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Finally, checking the notion that practitioners will prefer to learn about 
practitioner-oriented methods such as action research yielded the following results: 



Table 6: Practitioner Methods’ popularity and most popular methods by 
public/private school distinction, professional category (% answered 
affirmatively) 




Practitioner 

Method 

(Action 

Research) 

Popular? 


Most Popular Methods Listed 


Public School 
Background 


44% 


Qualitative methods generally; 
Action research; Historical methods 


Private School 
Background 


35% 


Qualitative methods generally; 
Interrogative research; Survey 
research 


Teachers 


40% 


Qualitative methods generally; 
Ethnography; Interrogative research 


Administrators 


10% 


Qualitative methods generally; 
Quantitative methods generally 


Teacher/ Administrators 


50% 


Qualitative methods generally. 
Action research; Content analysis 


Other Professionals 


66% 


Qualitative methods generally; 
Action research; Interrogative 
Research 



The respondents’ relative lack of interest in practitioner-oriented methods is 
somewhat surprising. With the exception of the other professional category, 50% or less 
of all the students indicated a particular interest in Action Research. Action Research 
appeared as a top choice among public school background students, among 
teacher/administrators and other professionals, but was not a specified top choice for 
other student groups. However, the students overwhelmingly indicate general interest in 
using qualitative methods. Is this a function of these particular students’ general 
preference for non-quantitative work and lack of comfort with statistically-based 
methods, or a reflection of the instructor’s own proclivities and biases toward certain 
kinds of methods? Unfortunately, the data do not give us any straight answers, although I 
suspect that both played a role in these results. 

Conclusion 



Making the case for the relevant, yet comprehensive coverage of research methods 
is tough in the face of resistance from the future education leaders. The findings of this 
paper focus the spotlight on several issues including: 
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• The challenges of scope and breadth in designing introductory methods 
courses; 

• The fact that some popular textbooks do not confront the problem of 
relevance very well; 

• The gap between faculty and student perceptions of key competences 
needed in research methodology; 

• The benefits of potentially incorporating more case studies and hands on 
instructional experiences for graduate-level introductory methods course 
as a means of better engaging educational leaders. 

In light of the challenges facing educational leaders and their use of education 
research, now more than ever, delivering well thought-out curriculum on research 
methodologies is an important goal for Schools of Education. This paper is only a first 
attempt at laying out the general issues and concerns surrounding an increasingly 
important and pertinent feature of graduate training in education. Hindered in 
generalizability by its use of data from only two programs at a single institution and a 
small but purposive sample of textbooks, this paper cannot identify best practices or 
identify the best resources available to instructors. To accomplish that goal, further 
research is called for that can incorporate comparisons of similar courses taught at a 
variety of Schools of Education, accompanied by faculty interviews and student surveys 
as well as a more comprehensive comparison of available textbooks. 
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Research Utilization; Instructional Design/Development; Leadership; Teacher 
Educational Development 



Appendix; Student Survey 



This survey should take approximately 10 minutes to complete. 
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Part I. 

What is your gender? Male Female 

What is your age? 

What professional category do you feel describes you best? (Please check one) 

Teacher Administrator Other 

How long have you been a teacher or administrator (please indicate number of 
years) ? 



If you are a teacher, what subject do you teach? 

If you are a teacher, what grade level do you teach? 

If you are an administrator, what is your title? 

If you are an administrator, do you work at the building, district or county level? 



Building District County 

Please list your previous postsecondary academic degrees: 




Have you taken previous courses in research methods? Yes No 




Part II. 
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On a scale of 0 to 4, with 4 being the highest score possible, how useful did you 
find the following elements of the Introduction to Research Methods course in 
support of your understanding of education research? Circle the most 
appropriate score. 



1 . Journal exercise 0 

2. Library resources guest lecture 0 

3. Assigned readings 0 

4. Textbook 0 

5. Class handouts 0 

6. Class lectures 0 

7. Class discussions 0 

8. Group article critique 0 

9. Interview simulation exercise 0 

10. Research proposal exercise 0 

11. Individual article critique 0 

12. Other (please indicate) 



12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

0 12 3 4 



Part III. 

1. In light of the information you have been given in class about different 
qualitative and quantitative methods, have you developed a preference for 
using a particular class of methods or research techniques in your future 
work, either as a producer or consumer or research? 

Yes No 

1 .b If you answered yes, check all the responses that apply to you. 

I have developed a preference for potentially using: 
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Qualitative methods in general 

Ethnography 

Ethnographic Methods 

Historical Research 

Interrogative Research (Interviews, Focus Groups) 

^Action Research 

Content Analysis 

Discourse Analysis 

Quantitative methods in general 

Experimental Research 

Quasi-experimental Research 

^Associational Research (Correlational, Causal-Comparative) 

Survey Research 

2. Have you developed an interest in learning more about particular research 
methods and techniques? Yes No 

2.b If you answered yes, which of the following would you like to learn 
more about? (Please check all that apply.) 

Qualitative methods in general 

Ethnography 

Ethnographic Methods 

Historical Research 

Interrogative Research (Interviews, Focus Groups) 

^Action Research 




22 



2 1 Rough draft - Not for citation without permission of the author 



Content Analysis 
Discourse Analysis 
Quantitative methods in general 
Experimental Research 
Quasi-experimental Research 

Associational Research (Correlational, Causal-Comparative) 

Survey Research 

Other (Please specify below) 



2.C If you answered yes, would you be interested in taking a course that 
focuses on the methodologies and/or techniques that you have listed? 

Yes No 



2d. If you answered yes, can you comment below on what about these 
methods and techniques are attractive to you as either a producer or 
consumer of research? 



Thank you for your assistance! 
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